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Abstract—Image fusion aims to retrieve information data 
from a set of multiple images into a composite output image. 
Pan-sharpen is the method of combining a high spatial 
resolution panchromatic (PAN) image and a high spectral 
resolution multispectral (MS) data of the same scene. It is not 
only beneficial for enhancing the spatial contents but also to 
preserve its spectral quality to be more useful for 
interpretation. Intensity-Hue-Saturation (IHS) transform is the 
most commonly performed method of image fusion, but it 
causes some spectral distortion. A proposed hybrid approach 
applies IHS fusion with a Stationary Wavelet Transform 
(SWT) because the recent usually preserves the color 
information via multilevel decomposition. By using different 
fusion rules, for combining the corresponding approximation 
and details coefficients obtained from wavelet decompositions, 
a coefficient combining and activity level measurement based 
fusion rules are executed. The main characteristic of the 
composite platform is utilizing the features of the combined 
algorithms with different fusion rules into a single image. The 
fusion results are evaluated and compared with results by 
conventional fusion techniques. Choosing of the optimum 
fusion rule counts on the type of imageries to be fused and 
depending upon the application. Visual inspection and 
quantitative analysis demonstrate that successful results 
among the different fusion methods. 


Ki eywords— Multi-sensor image fusion; PCA; IHS; DWT; SWT; 
Filter mask 


I. INTRODUCTION 


Multi-sensor data fusion is a challenging research in many 
remote sensing (RS) image processing applications because 
of the different characteristics of remotely sensed data 
acquired from multisensory [1]. For satellite RS 
applications, Image fusion process is the most popular 
approach of combining a single band PAN or synthetic 
aperture radar (SAR) image with high spatial resolution and 
MS (color) data with a lower spatial resolution in order to 
create a resultant unique image with more significant and 
accurate information than the individual sources [2]. The 
fusion techniques should ensure that all spatial details and 
spectral characteristics in the original images are injected 
into the fused image, without introducing inconsistencies or 
any artifacts, which may affect the quality of the output 
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image [3]. Therefore, the new fused image has to be more 
informative and spatially better than any of the individual 
images and doesn’t lead to wrong diagnosis. 

According to the fusion of multi-sensor images, there is 
generally a compromising between getting a good image, 
spatially and spectrally, which resulting in information loss 
[4]. In general, fusion takes place at three different levels, 
they are pixel level (lowest level), feature level 
(intermediate level) and decision level (high-level) fusion. 
In this study, a data/pixel level based image fusion is 
presented in order to perform a fusion process, generating a 
single complementary image, containing an extra truthful 
description than any of the single sources [5]. The most 
common approaches for image fusion are the band 
substitution methods including the Intensity-Hue-Saturation 
(IHS) and Principal Components Analysis (PCA), etc. 
Those approaches; are generally known as component 
substitution (CS) algorithms, this group of methods 
generates a high spectral distortion to the original color 
information [6]. Contrarily, the multi-resolution approach 
has been proposed for image fusion which introduces the 
multi-scale decomposition (MSD) such as the basic Discrete 
Wavelet Transform (DWT) [21, 23]. Although the wavelet 
analysis is an interesting tool which can avoid information 
distortion and getting better performance for image fusion. 
the main disadvantage of DWT is a shift-variant. Therefore, 
this disadvantage can be overcome by using a wavelet 
technique which is shift-invariant wavelets is called 
Stationary Wavelet Transform (SWT) [1-21-22]. 

However, many comparisons have showed that IHS 
fusion results present a good color-feature integration, but 
there is a big color distortion. Wavelet based fusion methods 
demonstrates color information preserving better than does 
the IHS. Researchers have developed hybrid fusion 
algorithms, integrating two or more fusion techniques in one 
framework to solve the color distortion problem. Several 
fusion rules have been submitted for image fusion which is 
either activity-level measurement method or coefficient 
combining method [19-20]. Two methods are being 
discussed. 


This paper proposed a novel hybrid IHS-SWT based 
image fusion algorithm, integrating both coefficient 
combining Based; choose-mean for approximation 
coefficients and activity level measurement; Window Based 
Activity (WBA) fusion rules such as High-Pass (HP), Low- 
Pass (LP) and Weighted Average (WA) filter masking for 
details coefficients in an individual single fused images. 

For estimation the quality of fused products, a qualitative 
and quantitative evaluation is presented. Results 
demonstrate that the proposed integrated algorithm 
improves the quality of Quick Bird fused images in 
comparison of the traditional methods. 

The paper is summarized as follows: Section II gives a 
brief review about the classical image fusion techniques. 
Section HI is devoted to the proposed hybrid algorithm. 
Section IV deals with the experimental results and describes 
quality metrics for evaluation. Finally, conclusions 
including discussion are recapitulated in section V. 


II. | CONVENTIONAL FUSION TECHNIQUES 


High resolution PAN and coarser MS imageries have to 
be pre-processed before pan-sharpening. Pre-processing 
may involve Image Registration (IR) to make input images 
aligned and registered which is a prerequisite to apply 
fusion techniques. It is followed by Resampling (RS) of 
multispectral image to that of PAN image. Histogram 
matching is performed before applying pan-sharpen. In this 
step, histogram matching is utilized for equalization 
standard deviation and mean to make the spectral distortion 
less for both images and for solving the problem of non- 
linear differences between them [16]. The next section 
describes some of the traditional image fusion Methods for 
satellite images. 


A. Component Substitution (CS)Methods 


The principal of CS (also called projection substitution) 
techniques is to project the original MS bands using a 
transformation in different components. Then, one of the 
resulting components which contains the spatial information 
is replaced by the higher spatial resolution pan imagery. The 
higher correlation between both input data, the less 
distortion of data. Thus, by applying the inverse transform, 
the fused image is produced [29]. 


1) Intensity-hue-saturation (IHS) Fusion method: 


Intensity, hue and saturation technique is a standard 
procedure in pan-sharpening approaches in the CS category. 
To understand IHS, the three primary bands (red, green and 
blue) of the MS imagery have to be converted into (IHS) 
space. In IHS color space, the spatial details are enclosed in 
intensity component, while the spectral information of a 
color image mainly lies in hue and saturation. The histogram 
of the Pan imagery is usually matched to the intensity (I) 
band before the replacement in order to get a better fusion 
quality. Afterwards, the (I) component is substituted by the 
high-resolution panchromatic imagery then the histogram 
matched Pan is gathered with H and S bands and finally 
applying a reverse transformation from IHS to RGB space, 
generating a high spatial color image. Fig. 1. Describes IHS 
method [30-31]. 
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Fig. 1. Image Fusion with the IHS method. 


If the correlation between PAN and MS images is 
very high, then the fused image has better spatial resolution 
for analysis. The main disadvantage of CS based techniques 
is that significant spectral distortion which is also called 
radiometric distortion [11]. 


B. Multi-Resolution analysis (MRA) 


The MRA methods include Laplacian pyramid and the 
wavelet transform is based on MSD of the high spatial 
resolution imagery to extract the spatial details and then 
transferring into the color image. Particularly, The MRA 
fusion approaches usually give superior performance in 
terms of reducing the spectral distortion compared to CS 
based methods [12, 32]. 


1) Stationary Wavelet Transform: 


Stationary wavelet transform (SWT) is an 
approximation of the classical DWT. Whereas, SWT is 
slightly different from the shift-variant DWT, also known as 
UN decimated DWT or shift-invariant because it is carried 
out by suppressing the step of down sampling at each scale 
instead of up-sampling the filters at each scale in the DWT 
computation. In SWT, filter coefficients are modified by 
zeroes insertion. [23-33]. 


Standard DWT is shift-variant which causes artifacts in 
the fused images, therefore it can be avoided by applying 
SWT [25]. The 2D-SWT solves the decimation problem by 
applying modified High pass and low pass filters at each 
level without any decimation. SWT produces two sequences 
at the next stage. The two new sequences are having same 
length as that of the source image. SWT is schematically 
visualized in Fig. 2. [27]. 
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Fig. 2. 2-D Stationart Wavelet Transform. 


In order to implement SWT on an image, the dimensions 
of the image should be integer multiples of 2] , where jJ 
represents the number of wavelet decomposition levels 
applied. The LP and HP filters employed for wavelet 
decomposition at j-th scale are represented by the symbols 
hiş and hiy respectively. The outputs of LP and HP filters are 
corresponding to the approximations (low frequency) and 
details (high frequency) sub images of the input image 
respectively. Therefore, the SWT produces four sub-images 
at a lower scale; Approximation (A), Horizontal (H), 
Vertical (V) and diagonal (D). 

a) Fusion Rules. 


Image fusion rules can be defined by the strategy 
of the fusion, which determines how the different 
coefficients will be combined [18]. Image fusion rules are 
divided into two main parts, coefficient combining method 
and activity-level measurement method. Choosing MAX or 
MIN, or MEAN fusion rules are the most popular 
coefficient combining methods. Whereas, three methods are 
generally for adopting the activity level of a MSD; 
coefficient-based (CB), window-based (WB) and region- 
based (RB) which requires a good region segmentation. 
Therefore, it is ineffective for some applications [8-9]. 
Window based activity level measurement (WBA) fusion 
rule considered that there is a high correlation among 
neighboring pixels, for example, 3x3 or 5x5 windows. Many 
options of WBA, such as a weighted average (WA-WBA), 
rank filter and the space frequency [9, 19]. 

According to these mathematical fusion rules, the 
decomposed wavelet coefficients extracted from the source 
images are merged to form fused wavelet coefficients before 
the backward wavelet transform. Test data are decomposed 
using multi-level 2-D SWT. Approximation coefficients 
(APX) are subjected to mean selection as a pixel based 
fusion rule. 

Where A (i, J) and B (1, J) are the input images and M (i, J) 
is the output fused approximation sub image. While 3x3 
square high-pass, low-pass and weighted average filter 
masks are applied to detail coefficients (DET) as a WB 
fusion rules, individually, as follows: 
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MHI. PROPOSED ALGORITHM 


As discussed before, exploiting the wavelet analysis 
integrated with IHS Transform is to enhance the advantages 
and overcome the limitations of each technique. Combining 
different fusion rules, leads to developing hybrid algorithm, 
integrates the merits of the different fusion rules for fusing 
the decomposed low and high frequency sub images. Fig. 3. 
Shows the flow chart of the algorithm that divided into three 
main procedures: 


(1) IHS Transform. 
(ID Multi-level 2-D Stationary Wavelet Transform. 

(IID) Applying the different fusion rules for combining low 
and high frequency coefficients of wavelet 
decomposition. 

1) Algorithm: 


Before the image fusion, the MS image are co- 
registered to the corresponding Pan image and resampled 
using Cubic Convolution to the pixel sizes of the Pan image. 
Cubic Convolution method reduces the stepped effect which 
produced by the Nearest Neighbor method .it takes more 
neighboring cells compared to bilinear resampling and it 
makes pixels looking smooth [34]. 


We have summarized the proposed algorithm steps as 

follows: 

1. Transfer the MS image from (RGB) color space into 
the IHS space (Forward IHS transform). 

2. PAN imagery is histogram matched with Intensity (1) 
component and generate a new pan image (Matched 
Pan). 

3. Perform independent multi-level 2-D SWT of the 
Matched Pan image and Intensity (I) to the level L to 
obtain the approximation (LL") and the detail (LH! 
„HL! ,HH!) coefficients up to level 2 of decomposition 
where | = 1, 2,..., L. 

4. Choosing mean pixel based algorithm for 
approximations (LL") by choosing the average valued 
pixels from the APX of the Matched Pan and I 
images. 


LL: = mean ((LL"1 (i, j)), (LL'» (i, j)) (4) 


Here LL’, is the fused and LL’; and LL*p are the 
approximations coefficients, 1 and j represent the 
positions of the sub-images pixels. 


5. Based on filter mask, A 3X3 window is subjected to 
the DET sub-images is applied. 

6. Perform window based fusion rule of detail 
coefficients by using three different square masks of 
3X3 filters. The masked coefficients are added from 
each sub-band as below. 


LH" = mask (LH!) + mask (LH!p) (5) 

HL!; = mask (HL!) + mask (HL') (6) 

HH! = mask (HH!) + mask (HH!p) (7) 
Where, 


LH", LH'p HL', HL'p, HH', HH'p are the vertical, 
horizontal and diagonal high frequencies respectively. 
LH HL; HH; are the fused detail sub-bands 
respectively. LL'; is the fused APX through mean rules 
and LH! HL'; HH’; are the vertical, horizontal and 
diagonal DET by applying the mask (WB) fusion rules 
for 1 =1,2.,...,L. 
7. By concatenating the output fused approximation and 
detailed coefficients, the fused result is reconstructed 
by inverse stationary wavelet transform (ISWT) 
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Fig. 3. HIS-SWT Flow chart fusion algorithm. 
2) Quality Metrics: 


However, neither qualitative nor quantitative 
evaluation alone will bring out satisfied results [15, 35]. 
Image fusion performance can be studied qualitatively by 
visual inspection of fusion results and quantitatively is done 
by statistically calculating and analyzing the assessment of 
image quality indexes as follows: 


ea 
Max* 


Peak Signal-to-Noise Ratio (PSNR) =:10 l0g10 isp) (8) 
Where, 
Max is the maximum pixel value of the given image. 
MSE is the mean square error. A higher PSNR value 
implies higher quality. 

Consider a (i, j) represents original panchromatic 
imagery, g (i, j, k) denotes the original multispectral 
imagery with resampled no of bands k to be the same size as 
the PAN image. While f (1, Jj, K) is the fused image with size 
M and N respectively. Root Mean Square Error (RMSE) is 
calculated as: 


a 
RMSE = Zio Eco (fG, j,k) — g(i j, k))?/MN (9) 


The spectral correlation coefficient (CC) is an index of 
similarity between the original MS and the fused images. 
The more closely of the CC to unity, identical the images 
will be. 
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The Entropy is the index of the richness of information 
content of image: 


CC (f, g) = (10) 


Entropy (E) = — 325° flogof; (11) 


The spatial quality is assessed by High pass filtering 
which is one of spatial indexes that can reflect the spatial 
fidelity of fused images as follows: 

-i -1 -1 
E g = (12) 
-i -1 -1l 

The correlation coefficient between the filtered PAN 

imagery and filtered merged results is used as an index of 


the spatial quality because most of spatial details are existed 
in the domain of high frequency [29]. 


Mask = 


IV. EXIPERMENTAL RESULTS AND EVALUATION 


Input data consist of co-registered Quick bird PAN image 
which is rendered as gray image and natural color MS image 
of ground spacing 0.6 and 2.4 meter with size of 512x512 
pixels for PAN image and 128x128 pixels for MS image. In 
Fig. 4, (a) is the high-resolution PAN imagery. (b), (c), (d) 
are the three primary colors (red, green and blue) of the MS 
imagery and (e) is the color image. 





Fig. 4. Original PAN and MS images. 


The tested methods include IHS, PCA, Standard 2D- 
DWT, multi-level 2D SWT, IHS-DWT, and the proposed 
IHS-SWT image fusion approaches as shown in Fig. 5. 
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Fig. 5. Fusion results obtained by different techniques: (a) Image Fusion by 
IHS. (b) Image Fusion by PCA. (c) Image Fusion by DWT. 
(d) Image Fusion by SWT. (e) Image Fusion by IHS-DWT. (f) Image 
Fusion by IHS-SWT. 


Fig. 5. Shows the different fused images using different 
pan-sharpen methods, as shown in above figures, a bit of 
color distortion has appeared in the fused image obtained by 
IHS method. And also, a significant spectral distortion 1s 
found in the fused image by using PCA. Although, the other 
fusion results are perceptually good, they have some spatial 
and spectral distortions, which are represented statistically 
in table I. The integrated IHS-SWT fusion method has the 
lowest MSE values and then the highest PSNR values. 
Despite the spectral quality is perceptually good, the edges 
are not well distinguishable. 


Note that The integrated IHS-SWT  pan-sharpening 
method has also been carried out using different sets of 
fusion rules between the APX and DET coefficients 1.e. 
Choose Max (CM)-WA,CM-LPF, CM-HPF, MEAN-WA, 
WA-MEAN, WA-CM,LPF-CM, HPF-CM. Results are 
evaluated for all possible combination methods for APX and 
DET coefficients. 

Fig. 6. Shows the output Results of using different sets of 
fusion rules for (APX, DET) coefficients. Table III presents 
the values of quality metrics for different set of fusion rules 
for APX and DET coefficients. 


TABLE I. Spectral assessment between image fusion techniques. 
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Table I presents the comparison between the spectral 
assessment of the fused results according to the different 
standard (standalone) approaches ; IHS, PCA, DWT and 
SWT and the hybrid fusion approaches; IHS-DWT and IHS- 
SWT, using different quality metrics at each band (B3, B2 
and B1) which represent the spectral bands (R,G and B). 
Quantitative analysis in table I shows that the hybrid IHS- 
SWT algorithm over performs the other methods in terms of 
values of the spectral quality metrics; E, CC, MSE and 
PSNR. 

Table II presents the values of the spatial CC between the 
original pan image and the different fused results obtained 
from the fusion techniques. It shows that the hybrid IHS- 
SWT results are not as high as compared to those obtained 
by applying other methods, such as PCA but still sufficient. 


TABLE II. Spatial assessment between image fusion techniques. 
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Fig. 6. Results of pan-sharpening using different sets of fusion rules 
(fusion rule for APX, fusion rule for DET). 


TABLE III. Spectral assessment between image fusion techniques combining different fusion rules (APX, DET). 
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As illustrated in Fig. 6. The results based on merging the 
coefficient combining and window based fusion rules. By 
adopting the MEAN-WA fusion rule which displays a good 
spectral quality and retains the boundary edges and the 
structural details without any inconsistencies to the fused 
image. Also MSE and PSNR values over perform the other 
values of different combinations. 

Using MEAN-WA fusion rule shows the highest values 
of the SCC compared with the other different combinations 
as shown in table IV. Adopting HP filter helps in improving 
the features of salient edges, but it causes some spectral 
distortions. Whereas, averaging or low-pass filter removes 
noise by taking the average values of window that 
surrounding the center pixel and weighted average filter has 
a less Blurring effect compared to Average filter. 
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TABLE IV. Spatial assessment between image fusion techniques 
combining different fusion rules (APX, DET). 
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V. CONCLUSION 


In this study, the work done helps in explaining the 
principles for further research in image fusion by wavelet 
transform that implements a shift-invariant SWT and 
preserves the spectral information via multilevel 
decomposition, integrating with IHS fusion algorithm and 
combining two different types of fusion rules in a resultant 
image. In addition, the IHS-SWT hybrid algorithm 
discussed provides promising results spectrally and spatially 
simultaneously than other methods, which may not have 
always a good detection of features. 
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The proposed architecture uses two different methods 
of fusion rules, choosing mean is a better choice for 
combining the approximations. Furthermore, applying 
window based fusion rules for details. The changes in 
performance of the applied fusion rules for test images 
clarify that the choosing of the most optimum fusion rule 
counts on the type of imageries to be fused. It provides 
using different masks for fusion, depending upon the 
application. 
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